Bandsaw: Log-Powered Test Scenario Generation for Distributed Systems

Software testing is a widely used technique to eliminate defects and improve software quality. Testing is especially useful in the context of large distributed systems, which are notoriously difficult to reason about formally and remain out of reach for many existing analysis tools. A key problem in formulating a test case is identifying scenarios that can be (1) induced by a valid execution of the system under test, and are (2) different from the scenarios exercised by the suite of existing test cases. Today, this is done manually — a developer writes the code for a test case after considering the system implementation and the existing test suite. For a distributed system, this mental effort can be overwhelming since test cases are usually concurrent, involve multiple nodes and numerous message interleavings. Moreover, when coming up with a new test case, the developer usually focuses on the code artifacts and rarely, if ever, thinks about the abstract scenarios that the testing code represents. The goal of our tool, Bandsaw, is to automate test scenario generation for distributed systems by considering the log of system’s test suite executions. Bandsawgenerated scenarios are intended to be converted into test cases by the developer, which can then be run to test the implementation. Bandsaw, therefore, saves the developer the mental effort involved in coming up with a new test case. Test scenarios are also more abstract and simpler than the underlying testing code. These scenarios help developers to more easily reason about their test suites. Bandsaw takes as input a console log file generated by the existing test suite, and outputs a new scenario. By construction, this scenario is (1) different from the ones already encoded in the test suite and (2) plausible — likely to be induced by a valid run of the system. Bandsaw generates scenarios by exploring different interleavings of concurrent events and stitching together previously observed scenarios at those points when the system is likely to be in a common state. The generated scenarios satisfy a set of automatically-mined, key temporal properties that are true of all the existing scenarios. The careful stitching of scenarios and the preservation of the mined properties make the resulting scenarios plausible. buy search search available